Skip to content

Validity and Stability Pipeline Development FS Benchmark#288

Closed
schaeferbasti wants to merge 49 commits intoautogluon:fe_benchmark_mainfrom
schaeferbasti:fe_benchmark_main_val_pipeline
Closed

Validity and Stability Pipeline Development FS Benchmark#288
schaeferbasti wants to merge 49 commits intoautogluon:fe_benchmark_mainfrom
schaeferbasti:fe_benchmark_main_val_pipeline

Conversation

@schaeferbasti
Copy link
Copy Markdown
Contributor

@schaeferbasti schaeferbasti commented Apr 10, 2026 •
edited
Loading

Issue #, if available:

Description of changes:

  • Edit feature_selection_benchmark_runner.py for validity and stability (fix bugs, make usable for cli, save results in csv files)
  • Soon: Add batch script for executing the runner with all datasets, methods and repeats

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

LennartPurucker and others added 16 commits April 9, 2026 17:18
@LennartPurucker
Change bool handling in new preprocessing (autogluon#287)
b78539e
@LennartPurucker
fix: correct mem estimate using X's features after transformation if …
d204116 
…needed, not internal ones
@Innixma
Add "datasets" filtering option
364fbef
@LennartPurucker
fix: num_repeats None bug for grouped data
d9f221d
@LennartPurucker
fix: sanity check problem type when loading data
9fd7344
@LennartPurucker
fix: also remap feature_metadata_in
f594fc8
@LennartPurucker
fix: user task arff checker edge case, more clear error
0185042
@schaeferbasti
fix: minor changes and corrections for the fs_benchmark_runner
1617453
@schaeferbasti
WIP fix: make data_foundry usable for now
9cd8bae
@schaeferbasti
WIP fix: save FeatureSelectionResult to csv
9712ff2
@schaeferbasti
fix: use usual constants for data_foundry_cache
013c06f
@schaeferbasti
maint: save results in results folder
ccf751a
@LennartPurucker
fix: renaming now on global level
7a8eb18
@LennartPurucker
add: basic dtype flags for task metadata
d0c688a
@schaeferbasti
maint: use args instead of download_data_foundry_dataset
bf6eb04
@schaeferbasti
maint: use args instead of download_data_foundry_dataset
9f22c32
@schaeferbasti schaeferbasti marked this pull request as ready for review April 10, 2026 11:18
schaeferbasti and others added 13 commits April 10, 2026 13:39
@schaeferbasti
maint: change path for cluster (need to check)
dc5ad57
@LennartPurucker
add: support for more dataset filters
b53dd54
@LennartPurucker
add: class-level cache and dynamic batch size
0185dde
@LennartPurucker
add: allow to only pass GPU for preprocessing
5c6e304
@LennartPurucker
unify logging path for parallel benchmarks
48c2cd2
@schaeferbasti
add: script for generating script for extra benchmark (needs to be te…
5ef21ed 
…sted)
@LennartPurucker
fix: allow missing classes in test split for grouped data
5754508
@Innixma
Add TabPrep configs
b61453d
@Innixma
Add fast task_metadata loading for EndToEnd.from_raw()
ebf772c
@LennartPurucker
Switch to cache-based text benchmarking (autogluon#289)
4263865
@LennartPurucker
maint: better logging for output of text cache
a783c90
@LennartPurucker
fix: ensure user tasks do not lose dtypes and go to pickle right away…
852457b 
… (instead of letting OpenML handle it)
@LennartPurucker
add: much more clever batching buckets
3b4c19c
@LennartPurucker
add: finalize text cache
53436a0
@LennartPurucker
maint: fix test for new default output size for stat text
9df4307
@LennartPurucker
fix: adjust folds to number of groups
9ffa667
@LennartPurucker
fix: do not load model if not needed
3efafd1
@LennartPurucker
fix: correct reasoning
f15e156
@LennartPurucker
fix: dtype wrong for splits
8f0b0d6
@LennartPurucker
add: filter future warnings from ray
55539cd
@LennartPurucker
add: memory est overhead for large data
e731335
@LennartPurucker
fix: better memory estimate for TabM and remove view usage
77c9639
@LennartPurucker
fix: realmlp cat codes
5a8b483
@LennartPurucker
fix: OOD cat codes support and nan handling
1c3d572
@LennartPurucker
add: support for lazy loading of X,y in TabArena and avoid x3 copies
9d1d6d1
@LennartPurucker
add: support disable retries
be2f763
@LennartPurucker
maint: make MLPs torch abstract models and adapt batch size for large…
63f9b16 
…r dataset usage
@LennartPurucker
maint: make TFMs torch abstract models
4208c3c
@LennartPurucker
add: support for setting max predict batch size
0f36c2f
LennartPurucker
LennartPurucker requested changes Apr 12, 2026
View reviewed changes
@schaeferbasti schaeferbasti deleted the fe_benchmark_main_val_pipeline branch April 13, 2026 09:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@LennartPurucker LennartPurucker LennartPurucker requested changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@schaeferbasti @LennartPurucker @Innixma